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Abstract 

This  paper  introduces  a  novel  Markov  process  formulation 
of  stochastic  fault  growth  modeling,  in  order  to  facilitate  the 
development  and  analysis  of  prognostics-based  control  adap¬ 
tation.  A  metric  representing  the  relative  deviation  between 
the  nominal  output  of  a  system  and  the  net  output  that  is  ac¬ 
tually  enacted  by  an  implemented  prognostics-based  control 
routine,  will  be  used  to  define  the  action  space  of  the  formu¬ 
lated  Markov  process.  The  state  space  of  the  Markov  pro¬ 
cess  will  be  defined  in  terms  of  an  abstracted  metric  repre¬ 
senting  the  relative  health  remaining  in  each  of  the  system’s 
components.  The  proposed  formulation  of  component  fault 
dynamics  will  conveniently  relate  feasible  system  output  per¬ 
formance  modifications  to  predictions  of  future  component 
health  deterioration. 

1.  Introduction 

Continuous  improvements  in  fault  diagnostic  and  fault 
growth  prognostic  technologies  have  spawned  a  prolific 
growth  in  the  application  domain  for  these  technologies,  and 
a  growing  research  interest  in  the  development  of  techniques 
for  using  available  diagnostic  and  prognostic  information  to 
improve  system  control  and  maintenance.  Presently,  a  clear 
analytical  process  may  be  applied  to  implement  and  evalu¬ 
ate  the  effectiveness  of  tools  for  fault  classification  and  fault 
growth  prediction;  however,  the  same  cannot  generally  be 
said  for  the  follow-on  task  of  making  intelligent  control  deci¬ 
sions  based  on  available  diagnostic  and  prognostic  informa¬ 
tion.  In  general,  the  application  of  an  analytical  approach  to 
the  implementation  and  evaluation  of  prognostics-based  deci¬ 
sion  making  techniques  will  be  complicated  by  the  potential 
for  high  uncertainty  in  estimating  the  future  effects  of  avail- 
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able  control  actions,  and  the  need  to  define  a  computationally 
tractable  space  of  present  and  future  control  decisions  to  be 
optimized  over. 

Several  recent  publications  have  suggested  metrics  for  quan¬ 
tifying  prognostic  uncertainty  and  evaluating  the  ability  of 
control  modifications  to  affect  meaningful  change  on  fault 
growth  predictions  (Saxena  et  al.,  2008;  P.  Wang  et  al.,  2012; 
Edwards  et  al.,  2010).  This  paper  describes  a  process  for  ab¬ 
stracting  uncertain  models  of  environmental  and  fault  growth 
dynamics  into  the  generalized  notation  of  a  non-deterministic 
Markov  process,  in  order  to  promote  an  application  indepen¬ 
dent  analysis  of  prognostics-based  control  strategies. 

Markov  decision  processes  have  been  widely  applied  to  the 
representation  of  problems  involving  decision  making  in 
the  presence  of  uncertain  or  stochastic  modeling  informa¬ 
tion  in  the  contexts  of  economics  (Hauriea  &  Moresino, 
2006),  supply  chain  management  (Parlara  et  al.,  1995),  sched¬ 
uled  maintenance  (Smilowitz  &  Madanat,  1994),  health  care 
(Sonnenberg  &  Beck,  1993),  and  many  other  disciplines,  in 
addition  to  being  a  widely  used  tool  for  describing  fault- 
adaptive  and  robust  control  problems  (Zhang  &  Jiang,  2008). 
A  formal  description  of  fault  growth  modeling  and  remain¬ 
ing  useful  life  (RUL)  estimation  in  terms  of  Markov  process 
models,  as  well  as  a  survey  of  similar  stochastic  modeling 
techniques,  are  given  in  Banjevic  and  Jardine  (2006). 

The  state  transition  probabilities  in  a  Markov  process  descrip¬ 
tion  of  fault  dynamics  may  be  chosen  to  approximate  an  an¬ 
alytical  formulation  of  a  stochastic  process,  such  as  a  Gaus¬ 
sian  process  model  of  fault  growth  dynamics,  as  described 
in  Sankararaman  et  al.  (2009);  alternatively,  state  transition 
probabilities  may  be  defined  purely  based  on  empirical  ob¬ 
servations  of  the  fault  growth  process,  as  is  the  case  with  hid¬ 
den  Markov  model  learning  techniques  (Baruah  &  Chinnam, 
2005),  or  they  may  be  derived  from  a  combination  of  a  priori 
and  empirical  information. 
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The  derivation  of  stochastic  fault  growth  models  for  a  partic¬ 
ular  application  is  not  specifically  addressed  in  this  publica¬ 
tion;  however,  an  effort  is  made  to  clearly  identify  how  vari¬ 
ous  sources  of  uncertain  modeling  information  would  be  in¬ 
corporated  into  the  Markov  process  representation  of  fault  dy¬ 
namics.  Arguments  are  provided  for  defining  the  state  space 
of  the  fault  growth  process  in  terms  of  a  metric  represent¬ 
ing  the  relative  health  of  system  components,  as  well  as  for 
defining  the  action  space  of  the  fault  growth  process  in  terms 
of  a  metric  representing  the  relative  deviation  between  the 
system’s  nominal  output  response  and  the  net  system  output 
that  is  actually  enacted  by  an  implemented  prognostics -based 
control  routine  at  each  control  time-step.  Operational  con¬ 
straints  on  minimum  acceptable  system  output  performance 
and  maximum  acceptable  fault  growth  risk  will  be  formulated 
as  bounds  on  the  action  space  available  to  implemented  con¬ 
trollers. 

2.  Generalized  stochastic  modeling  of  the 

FAULT  GROWTH  PROCESS 
Consider  a  discretized  fault  growth  process  of  the  form: 

7 i&Si,  Si  =  {ai,ai+Ai,ai+2Ah  ...,ai+miAi}  (1) 

p\  j{x  (k) ,  a  (, k ) ,  w  (. k))=P(ji  (k+l)  =  Sj\ 

7i  (k)=Si,a  =  a(k)  ,w  =  w(k)),  (2) 

Si,  Sj  G  Si,  a  ( k )  G  A  ( k ) ,  w  ( k )  G  W  (k) ,  k  G  N 

m 

Ylpli,J  =  1’  Vi  G  {0,1,..., m)  (3) 

3=0 

where  7 1  (k)  is  a  random  variable  representing  the  magnitude 
of  the  Zth  component  fault  mode  at  time-index  k.  Si  represents 
a  uniformly  quantized  state  space  of  potential  fault  magni¬ 
tudes,  and  p\  :j(x,  a,  w)  represents  the  probability  of  transi¬ 
tioning  from  damage  state  s,;  to  damage  state  Sj,  given  esti¬ 
mates  of  the  system  state  x,  a  set  of  low-level  control  com¬ 
mands  a,  and  the  states  of  environmental  and  other  exoge¬ 
nous  inputs  to  the  system  w.  Eq.  (3)  specifies  that  the  sum  of 
all  transition  probabilities  defined  at  each  system  state  must 
always  be  equal  to  one. 

The  Markov  process  notation  given  in  Eqs.  (l)-(3)  may  be 
used  to  describe  fault  growth  processes  for  all  systems  in 
which  the  following  assumptions  are  satisfied: 

Assumption  1.  State  transition  probabilities  p\  •  are  only 
dependent  on  the  current  states  of  7 j,  x,  a ,  and  w,  and  not 
on  any  states  or  inputs  occurring  previously.  This  is  referred 
to  as  the  memoryless  assumption,  or  the  Markov  assumption. 
For  cases  in  which  the  fault  growth  process  is  not  completely 
memoryless,  but  future  states  only  depend  on  a  finite  number 
m  of  previous  states,  the  Markov  assumption  can  be  satisfied 
by  defining  the  state  space  of  the  process  to  be  the  ordered  m- 


tuple  of  the  current  state  and  the  m  previously  visited  states 
(H.  S.  Wang  &  Chang,  1996). 

Assumption  2.  State  transition  probabilities  are  considered 
to  be  time  invariant;  although,  it  may  be  the  case  that  fault 
growth  models  are  not  precisely  known  a  priori  and  must 
be  adapted  online  using  techniques  such  as  particle  filtering 
(Orchard  et  ah,  2008)  or  Bayesian  learning  (Saha  et  ah,  2009). 
Online  adaptation  of  the  prognostic  model  on  the  basis  of  past 
observations  will  technically  violate  Assumption  1 ;  however, 
the  error  induced  by  this  effect  is  typically  ignored  because 
model  adaptation  transients  are  generally  difficult  to  charac¬ 
terize  and  they  will  die  out  as  the  model  is  adapted. 

Assumption  3.  At  all  discrete  time-steps,  the  state  space 
S',  the  action  space  A ,  and  the  space  of  environmental  and 
other  exogenous  inputs  to  the  system  W  are  adequately  rep¬ 
resented  by  a  finite  set  of  states,  which  will  be  bounded  from 
above  by  the  availability  of  computational  resources.  In  the 
event  that  fault  growth  must  be  modeled  as  a  continuous  time 
process,  such  as  the  Poisson  process  (Shetty  et  ah,  2008),  a 
representation  of  fault  growth  modeling  similar  to  that  given 
in  Eq.  (2)  may  be  expressed  in  terms  of  a  continuous  time 
Markov  process  (Seifozo,  1979)  or  a  semi-Markov  process 
(Dong  &  He,  2007). 

The  required  assumptions  are  mild  enough  to  allow  a  wide 
array  of  fault  growth  processes  to  be  described  in  terms  of 
the  Markov  process  notation  given  in  Eqs.  (l)-(3)  (Guidaa  & 
Pulcini,  2011;  Tang  et  ah,  2009). 

2.1.  Formulating  the  Markov  process  in  terms  of  compo¬ 
nent  health  rather  than  fault  magnitude 

Component  fault  magnitudes  can  generally  be  described  by  a 
real  number  corresponding  to  a  measurable  physical  property 
such  as  crack  length,  spall  width,  or  pitting  depth;  although, 
in  many  cases,  faults  cannot  be  directly  measured  in  situ  and 
diagnostic  routines  are  needed  to  approximate  current  fault 
magnitudes  based  on  the  secondary  effects  observed  in  avail¬ 
able  sensor  measurements.  Sensor  noise  and  modeling  uncer¬ 
tainties  will  often  result  in  significant  diagnostic  uncertainty, 
and  it  is  common  practice  for  diagnostic  estimates  to  be  re¬ 
ported  in  terms  of  a  probability  distribution  over  the  potential 
fault  magnitudes  that  could  correspond  to  a  given  set  obser¬ 
vations. 

The  notation  introduced  in  this  subsection  will  add  a  layer 
of  abstraction  to  the  fault  magnitude  estimates  produced  by 
online  fault  diagnostic  and  fault  growth  prognostic  routines. 
Rather  than  formulating  the  Markov  process  in  terms  of  com¬ 
ponent  fault  magnitudes,  a  transformation  is  assumed  to  exist 
that  will  express  the  fault  growth  process  in  terms  of  an  ap¬ 
plication  independent  metric  representing  the  state  of  heath 
(SOH)  of  each  system  component. 

The  SOH  for  component  l  at  time  t  will  be  represented  in 
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terms  of  a  probability  distribution  over  a  uniformly  quantized 
set  of  component  health  percentages  between  0%  and  100%; 

7i  (t)  GS,  S  =  {Sl,  S2,  Sm}, 

si  =  0%,  Si  =  Si—  i  +  A%,  1  <  i  <  100%/ (4) 

where  A%  is  a  quantization  step-size  for  the  state  space  of 
7 1.  The  notation  given  in  Eq.  (2)  is  now  considered  to  define 
the  probability  of  transitioning  from  one  component  SOH  to 
another,  on  the  basis  of  the  current  states  of  at,  a,  and  w  at 
time-index  k. 

Component  fault  magnitudes  are  generally  expected  to  mono- 
tonically  increase  with  time;  correspondingly,  component 
SOH  should  monotonically  decrease  with  time.  A  mandate  of 
monotonically  decreasing  component  health  is  incorporated 
into  the  Markov  process  notation  as  follows: 

m 

Pij  =  0,  if  j  >  i,  Y,Pi,i  =  1  G  {°’  1> 
o= o 

This  constraint  will  be  problematic  for  techniques  that  model 
process  uncertainty  with  an  analytical  distribution  that  lacks 
an  explicit  lower  bound.  For  example,  in  the  case  of  fault 
growth  prognostic  techniques  such  as  Kalman  filtering  and 
Gaussian  process  modeling,  an  assumption  of  Gaussian  un¬ 
certainty  will  introduce  some  probability  that  the  fault  mode 
does  not  monotonically  increase.  In  such  cases,  it  would  be 
necessary  to  assure  that  the  probability  attributed  to  the  non- 
realizable  outcomes,  P  (7;  (r)  <  7 1  (t))  for  r  >  t,  will  be 
acceptably  small.  It  will  not  always  be  the  case  that  com¬ 
ponent  health  is  strictly  monotonically  decreasing;  however, 
consideration  of  this  constraint  serves  to  illustrate  the  flexibil¬ 
ity  provided  by  representing  modeling  uncertainties  in  terms 
of  a  quantized  probability  mass  function  fpmf),  when  com¬ 
pared  to  techniques  that  assume  a  continuous  probability  dis¬ 
tribution  function  (pdf). 

2.2.  Formulating  the  action  space  of  the  fault  growth  pro¬ 
cess  in  terms  of  commanded  and  applied  loads 

Consider  a  process  model  for  component  damage  accumula¬ 
tion  that  is  expressed  in  terms  of  a  metric  representing  the 
load  or  stress  applied  to  a  component  at  each  control  time- 
step: 

Pli,j{ui)  =  P(n  (fc+1)  =  sil7z  (■ k)  =  Si,ui=ui  (k)), 

Si,Sj  G  S,  Ui  €  Ui  (fc)  (6) 

where  ui  represents  the  load  applied  to  component  l  and 
Ui  ( k )  represents  the  domain  of  feasible  component  load  al¬ 
locations  for  component  l  at  time-index  k.  The  component 
loading  variable  may  represent  pressure,  force,  torque,  or  a 
wide  variety  of  other  stressors  that  drive  component  damage. 

Determination  of  Ui  (k)  at  present  and  future  control  time- 


steps  will  require  a  mapping  function  to  translate  system 
hardware  limitations  and  estimated  environmental  loading 
conditions  into  the  component  loading  domain.  A  mapping 
between  available  low-level  control  actions  and  feasible  com¬ 
ponent  loadings,  as  well  as  an  inverse  mapping  are  both  as¬ 
sumed  to  be  known: 

F  (a  (k) ,  at  (k) ,  7  ( k ) ,  w  (k))  :  A(k)  U  ( k ) 

-1  (7) 

F  (u(k)  ,x(k)  ,~f(k)  ,w(k))  :  U  (k)  -A  A(k) 

where  F  represents  a  mapping  from  a  given  system  state, 
health  state,  exogenous  demand  state,  and  a  set  of  low-level 
control  actions  available  at  time-index  k,  onto  the  domain  of 
feasible  component  load  allocations  available  at  time-index 

k. 

2.2.1.  Performance  constraints 

In  addition  to  being  defined  in  terms  of  the  hardware  limita¬ 
tions  of  control  effectors,  the  domain  of  feasible  component 
load  allocations  will  generally  also  be  bounded  by  operational 
constraints  on  minimum  allowable  output  performance  and 
maximum  allowable  fault  growth  risk. 

Consider  a  constraint  on  minimum  allowable  system  perfor¬ 
mance  that  is  defined  in  terms  of  a  maximum  allowable  devi¬ 
ation  from  some  commanded  system  state: 

|xc  (k)  -  x(fc)|i  <  Aj  (k) ,  i  e  {1,2,  (8) 

where  x  and  xc  are  n  dimensional  vectors  that  represent  the 
actual  and  commanded  output  states  of  a  system  respectively, 
and  A*  specifies  a  maximum  acceptable  error  between  the  ith 
dimensions  of  x  and  xc. 

If  the  system’s  kinematics  are  known,  then  Newton’s  laws  of 
motion  can  be  applied  to  express  the  system’s  dynamics  in 
terms  of  the  instantaneous  forces  on  its  constituent  compo¬ 
nents, 

x  =  f{x,u,w)  (9) 

If  the  system  is  overactuated,  then  formulating  the  sys¬ 
tem’s  dynamics  in  terms  of  instantaneous  component  loads 
will  result  in  actuation  redundancies  being  identified  by  the 
nullspace  of  u.  Consider  the  following  generic  representa¬ 
tion  of  nonlinear  system  dynamics: 

x  =  A  (x,  w)  +  B  (x)  u  (10) 

where  A(x,w)  €  K",  B  (x)  €  R"X9,  x(t)  €  ffi",  and 

u  (t)  £  K9.  If  B  (x)  does  not  have  full  column  rank,  i.e, 
rank{5  (at)}  =  p  <  q  Vat,  then  the  system  is  overactuated, 
and  B  (at)  can  be  factorized  as: 

B  (at)  =  Bv  (x)  Bu  (x)  (11) 

where  B„  (at)  €  R"xp  and  Bu  (at)  6  Wxi  both  have  rankp. 
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The  system  can  now  be  rewritten  as: 

x=  A  (x,  w)  +  Bv  (x)  v 
v  =  Bu  (x)  u 

where  u(t)  £  Mp  represents  the  net  output  control  effort  pro¬ 
duced  by  the  system’s  q  components. 

Inverting  the  dynamics  given  in  Eq.  (12)  enables  the  perfor¬ 
mance  constraint  given  in  Eq.  (8)  to  be  expressed  as  a  max¬ 
imum  allowable  deviation  from  a  given  output  control  effort 
profile: 

r  =  Bv  (xc)~1j  (xc  -  A  (xc,  w)) 

(fe)  -  n  (fc)|  <  Aj  (k) ,  i  £  {1, 2, 

where  v  and  r  are  p  dimensional  vectors  that  represent  an  ac¬ 
tual  and  a  desired  net  output  force  to  be  exerted  by  the  system 
at  a  given  time,  and  Aj  (k)  specifies  a  maximum  acceptable 
error  between  the  ith  dimensions  of  v  and  r. 


health 


Figure  1.  Uncertainty  in  fault  growth  predictions  for  high  and 
low  loads  at  various  prognostic  horizons 


components.  In  such  cases,  the  production  of  prognostic  es¬ 
timates  with  the  highest  degree  of  realism  will  require  future 
component  loadings  to  be  modeled  as  a  stochastic  process 
that  accounts  for  the  statistics  of  all  parameters  affecting  com¬ 
ponent  load  allocations  within  the  controlled  system. 


Because  Bu  ( x )  has  full  column  rank,  a  system  response,  x, 
is  uniquely  determined  by  however,  if  the  system  is 

overactuated,  then  the  allocation  of  load  among  functionally 
redundant  components  may  be  specified  by  minimizing  the 
aggregate  component  damages  corresponding  to  load  alloca¬ 
tions  in  the  nullspace  of  Bu  (x).  Prognostics-based  control 
in  terms  of  component  load  allocations  has  been  analyzed 
for  an  overactuated  electro-mechanical  actuator  and  an  un¬ 
manned  ground  vehicle  in  previous  publications  (Bole  et  al., 
2010,2011). 

2.2.2.  Prognostic  constraints 

Prognostic  constraints  are  typically  specified  in  terms  of  a 
lower  bound  on  the  failure  time  of  system  components.  Con¬ 
straints  on  the  minimum  acceptable  component  failure  time 
may  be  specified  in  terms  of  a  maximum  acceptable  prob¬ 
ability  that  the  component  will  reach  0%  health  by  a  given 
time: 

P  (7 1  (' *m )  =  0%|7 1  ( tp ) ,  w  ( tp ) ,  ui  ( tp ))  <  on  (14) 

where  tp  is  the  time  at  which  the  fault  growth  prediction  is 
made  and  oti  is  an  upper  bound  on  the  probability  that  com¬ 
ponent  l  is  failed  at  time  £m- 

If  U[  was  known  over  the  domain  t  =  [tp,  then  for¬ 
ward  induction  could  be  used  to  evaluate  Eq.  (14)  from  Eq. 
(6).  Many  publications  on  the  topic  of  prognostics-based  con¬ 
trol  opt  to  simplify  the  prognostics  problem  by  assuming  that 
component  loadings  will  be  unvarying  over  the  prediction 
horizon: 

u(t)=u(tp)  Vt£[tp,tM]  (15) 

However,  in  most  cases,  time-varying  environmental  load¬ 
ing  conditions  and  time-varying  component  health  estimates 
are  expected  to  result  in  time-varying  loadings  on  a  system’s 


Consider  the  drawing  in  Figure  1,  the  growth  of  uncertainty 
in  component  health  estimates  at  prognostic  horizons  of  in¬ 
creasing  length  is  shown  for  two  potential  loading  conditions, 
denoted  as  ‘high  load’  and  ‘low  load’.  As  shown  in  the  figure, 
fault  growth  estimates  at  different  loading  conditions  are  ex¬ 
pected  to  diverge  over  time.  In  cases  where  prognostic  uncer¬ 
tainty  becomes  very  large,  due  to  high  uncertainty  on  predic¬ 
tions  of  future  component  loading  conditions,  or  high  uncer¬ 
tainty  on  predictions  of  component  damage  as  a  function  of 
loading  profiles,  the  magnitude  of  prognostic  uncertainty  may 
be  managed  by  limiting  the  length  of  the  prediction  horizon. 

The  specification  of  a  lower  bound  on  acceptable  value  at 
risk  (VaR)  assessments  of  system  health  over  the  range  t  = 
[ip  Am]  may  be  used  to  enforce  constraints  on  component 
longevity,  using  fixed  horizon  prognostic  predictions.  The 
VaR  of  a  random  variable  X  at  a  confidence  level  ip  is  de¬ 
fined  as: 

VaR v,  (X)  =  inf  {x  £  R  :  P  {X  <  x)  >  tp}  (16) 
A  finite  horizon  prognosis  constraint  is  written  as  follows: 

VaRft  (7j  (tp  +  t))  >Ci(tp  +  r)  (17) 

where  r  specifies  a  time  horizon  at  which  prognostic  con¬ 
straints  will  be  evaluated,  ( ;  specifies  a  lower  bound  on  the 
depletion  of  component  health  over  the  range  t  =  \tp.  f  ;vf], 
and  fli  defines  the  maximum  acceptable  probability  that  the 
health  of  component  l  is  less  than  Q  at  time  tp  +  r  . 

An  example  definition  of  Q  (t)  is  given  in  Figure  1.  The  spec¬ 
ification  of  an  appropriate  profile  for  (1  (f)  will  be  left  as  a 
design  choice.  Some  general  considerations  for  the  specifica¬ 
tion  of  an  appropriate  profile  include: 

1.  If  component  health  is  expected  to  be  monotonically  de¬ 
creasing,  then  £/(£)  must  also  be  a  monotonically  de- 
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creasing  function.  Additionally,  if  it  is  ever  the  case  that 
VaRpt  (7 1  ( tp ))  <  (1  ( tp  +  r),  then  the  prognostic  con¬ 
strain  is  unsatisfiable. 

2.  The  greater  the  difference  between  VaRpt  (7 j  (■ tp ))  and 
0  (tp  +  T)-  the  greater  the  control  freedom  allowed  un¬ 
der  the  prognostic  constraint.  Online  or  a  priori  infor¬ 
mation  could  be  used  to  make  adjustments  to  Q  (t)  so  as 
to  allow  for  greater  control  freedom  during  higher  pri¬ 
ority  time-periods  at  the  expense  of  potentially  incurring 
greater  component  health  deterioration  over  those  time- 
periods. 

2.3.  Fault  prediction  in  terms  of  relative  deviations  from 
nominal  system  outputs 

Consider  the  existence  of  a  nominal  control  system,  which 
would  adequately  control  a  system  in  the  absence  of  any  com¬ 
ponent  degradation  modes.  This  section  introduces  a  met¬ 
ric  to  represent  the  relative  deviation  between  the  net  output 
control  effort  that  would  have  been  exerted  by  the  system  on 
its  environment  if  a  nominal  control  law  were  used,  and  the 
net  output  control  effort  that  is  actually  exerted  by  a  given 
prognostics-based  control  routine  at  each  control  time-step. 

The  proposed  metric  is  defined  for  each  of  the  system’s  output 
degrees  of  freedom  as: 

/)i(fc)  =  M  ;e{i,2,..,p}  (is) 

where  u*  and  v  represent  the  net  output  control  effort  that 
would  have  been  commanded  by  a  nominal  control  law  and 
the  net  control  effort  output  that  is  actually  commanded  by  an 
implemented  control  routine  at  time-index  k. 

The  Markov  process  defined  in  Eq.  (6)  is  rewritten  in  terms 
of  this  new  metric  as: 

Plsz,Sj(p(k))  =  p(-n  (fc+l)  =  Sjh  (k)  =  Si,p  =  p(k)) 

=  J  P(vr  (k)  | pr  =  pr  (k))  ■  (19) 

vre  Nr{k) 

P(li  (k+ 1)=  Sjl'yi  (k)  =  Si,  u  (k)=H(v  (k)))  du 
Si,Sj  G  S,ui  G  Ui  (k)  ,  r  G  {1,2, 

where  P(vr{k)\pr)  can  be  estimated  using  available 
stochastic  modeling  of  the  future  net  output  control  effort 
demands  on  the  system,  and  H  (v  (k))  represents  a  map¬ 
ping  from  a  net  system  output  force  vector  to  a  component 
loading  vector.  As  described  in  Section  2.2.1,  if  no  over¬ 
actuation  is  present  in  the  system,  then  component  loadings 
are  uniquely  specified  by  a  net  system  output  force  profile; 
however,  if  the  system  is  overactuated,  then  the  nullspace  of 
the  component  loading  domain  can  be  resolved  by  an  opti¬ 
mization  routine  that  seeks  to  minimize  the  aggregate  loss 


of  health  among  functionally  redundant  degrees  of  freedom. 
Note  that  all  modeling  of  internal  and  external  dynamics  that 
affect  component  degradation  are  now  described  by  the  prob¬ 
ability  transition  matrix  p's  s  (p).  This  formulation  of  fault 
growth  dynamics  provides  a  convenient  means  for  analyzing 
the  prognostics-based  control  problem,  because  it  directly  re¬ 
lates  modifications  to  system  output  performance  to  predic¬ 
tions  of  component  degradations. 

The  performance  constraint,  defined  in  terms  of  allowable  net 
system  output  control  effort  in  Eq.  (13),  is  now  expressed  in 
terms  of  p  as: 

|1  -  Pi  (k)\  <  A*  (k) ,  i  G  {1,2,  (20) 

where  A,  defines  a  constraint  on  the  maximum  allowable  de¬ 
viation  from  a  system’s  nominal  control  effort  output  in  di¬ 
mension  i  at  time-index  k. 

A  uniformly  quantized  state  space  for  p,  under  the  perfor¬ 
mance  constraint  is  defined  as: 

Pi  (k)  G  0(fc) ,  0(fc)  =  [91,d2,...,0m\, 

0i  =  1  —  A*  (k) ,  9m  =  l  +  A i(k)  (21) 

2.3.1.  An  example  of  output  control  effort  regulation 

A  simple  example  is  considered  here  to  examine  the  regula¬ 
tion  of  a  system’s  net  control  effort  output  using  the  perfor¬ 
mance  metric  p.  The  example  system  to  be  regulated  is  a 
linear  actuator  attached  to  a  simple  mass-spring-damper  sys¬ 
tem,  defined  by  a  mass  to,  a  spring  constant  k,  and  a  damping 
coefficient  c: 

nix  =  —kx  —  cx  +  v  (22) 

A  nominal  control  law  for  the  system  is  represented  by  the 
following  proportional  feedback  equation: 

v*  =  kp  ■  (x  —  xc)  (23) 

where  x  and  xc  represent  an  actual  and  a  commanded  actua¬ 
tor  position  respectively,  v  and  v*  represent  the  net  actuator 
output  force  and  that  commanded  by  the  nominal  control  law 
respectively,  and  kp  is  a  gain  coefficient  within  the  nominal 
controller.  Diagrams  of  the  mass-spring-damper  system  and 
the  proposed  control  law  are  shown  in  Figure  2.  Values  for 
all  variables  in  the  controlled  mass-spring-damper  system  are 
given  in  Table  1 . 

In  this  example,  the  net  output  force  exerted  by  the  controlled 
effectors  on  the  environment  and  the  performance  metric  used 
to  regulate  that  output  are  both  one  dimensional: 

v  =  pv*  (24) 

Figure  3  shows  the  behavior  of  the  system  as  p  is  linearly  var¬ 
ied  over  the  domain  [0.2, 1],  It  should  generally  be  expected 
that  smaller  values  of  p  will  induce  greater  errors  in  tracking 
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Plant  Model 


v(k)  r\V*(k) 


C 


|  p(k) 

Control  System 


e{k)r^x(k) 
xc(k) 


Figure  2.  Diagram  of  mass-spring-damper  plant  model  and 
control  system 


Coefficient 

Value 

m 

1 

k 

2 

c 

5 

kp 

15 

Table  1.  Coefficients  used  in  mass-spring-damper  simulation 


a  desired  output  profile,  and  these  induced  errors  will  result 
in  greater  net  actuator  output  force  demands  by  the  nominal 
control  law.  Both  of  these  features  are  clearly  visible  in  the 
experimental  results  given  in  Figure  3.  Another  observation 
that  can  be  made  from  the  results  given  in  Figure  3  is  that 
while  p  <  1  will  always  result  in  an  apparent  reduction  in  the 
net  output  control  effort  that  would  have  been  commanded  by 
a  nominal  control  law  at  a  given  control  time-step,  the  error 
dynamics  that  are  induced  by  lowering  p  will  not  necessarily 
result  in  lowered  net  system  output  control  effort  over  a  finite 
window  of  observation.  A  marginally  increasing  trend  in  the 
peak-to-peak  actuator  loads  over  a  cycle  is  observed  as  p  is 
lowered;  this  type  of  behavior  illustrates  the  fact  that  achiev¬ 
ing  a  good  tradeoff  between  the  reduction  of  component  load¬ 
ing  and  induced  errors  in  trajectory  tracking  will  generally 
require  closed  loop  regulation  of  p. 

3.  The  Prognostics-Based  Control  Problem 

The  prognostics-based  control  problem  can  generally  be 
viewed  as  an  optimization  problem,  in  which  implemented 
control  routines  will  select  values  of  pi  at  each  control  time- 
step  in  an  attempt  to  minimize  the  risks  posed  by  the  appli¬ 
cation  of  load  to  degrading  components,  while  also  minimiz¬ 
ing  any  deviation  from  a  system’s  nominal  performance.  The 
problem  of  specifying  appropriate  metrics  for  assessing  the 
risk  posed  by  probabilistic  prognostic  predictions  of  future 
component  health  deterioration  may  generally  be  considered 
independently  from  the  problem  of  building  prognostic  mod¬ 
els.  Assuming  that  discrete  Markov  modeling  will  provide 
a  sufficiently  accurate  approximation  of  a  system’s  fault  dy¬ 
namics,  then  the  Markov  process  notation  described  in  Sec¬ 
tion  2  could  be  used  to  evaluate  any  given  risk  metric  for  use 
in  any  given  prognostics-based  control  method.  Future  work 
will  address  the  specification  of  appropriate  risk  metrics  and 
the  use  of  this  Markov  process  formulation  for  deriving  and 
evaluating  prognostics-based  control  policies  on  sample  ap¬ 
plications. 


Preformance  Factor  Profile 


-5  -■ 


2  3  4 

Time  (minutes) 


Figure  3.  Plots  showing  position  profile  tracking  and  actuator 
loading  dynamics  as  p  is  varied  over  the  range  [0.2, 1] 


If  aversion  to  the  potential  degradation  of  a  system’s  nomi¬ 
nal  output  loading  performance  and  aversion  to  the  potential 
degradation  of  component  SOH  are  adequately  expressed  in 
terms  of  an  expectation  of  accumulated  state  transition  costs, 
then  the  search  for  a  control  policy  that  optimizes  a  stochas¬ 
tic  system  of  the  form  disclosed  in  Eq.  (19)  is  expressible 
as  a  Markov  decision  process  (MDP).  MDPs  are  commonly 
used  to  analyze  problems  involving  decision  making  in  the 
presence  of  uncertain  or  stochastic  information,  and  opti¬ 
mizing  control  policies  may  be  identified  using  well  studied 
MDP  optimization  techniques  such  as  backwards  induction 
for  finite  horizon  optimization  problems,  and  linear  program¬ 
ming,  value  iteration,  and  policy  iteration  for  discounted  and 
average-reward  infinite  horizon  optimization  problems.  The 
requirement  that  fault  risk  must  be  expressed  as  an  expecta¬ 
tion  of  accumulated  state  transition  costs  over  a  finite  or  in¬ 
finite  horizon  may  seem  to  be  an  overly  restrictive  constraint 
on  the  general  problem  of  quantifying  risk;  however,  over  the 
past  several  decades  much  has  been  published  on  the  theory 
of  encoding  various  forms  of  risk  aversion  into  the  specifi¬ 
cation  of  MDP  state  transition  costs  (Hernandez  &  Marcus, 
1996;  Ruszczyriski,  2009). 

4.  Conclusions 

This  paper  introduced  a  novel  Markov  process  representation 
of  component  health  dynamics.  A  metric  representing  the 
relative  deviation  between  instantaneous  samples  of  the  net 
output  force  that  would  have  been  enacted  by  a  nominally 
controlled  system,  and  the  net  system  outputs  that  are  actually 
enacted  by  an  implemented  prognostics-based  control  routine 
at  each  time-step,  was  used  to  define  the  action  space  of  the 
Markov  process.  The  state  space  of  the  proposed  Markov  pro¬ 
cess  formulation  was  defined  in  terms  of  an  abstracted  met¬ 
ric  representing  the  relative  health  remaining  in  each  of  the 
system’s  components.  Operational  constraints  on  minimum 
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acceptable  system  output  performance  and  maximum  accept¬ 
able  fault  growth  risk  were  formalized,  and  the  mappings  nec¬ 
essary  to  impose  those  constraints  on  the  domain  of  feasible 
control  actions  available  to  implemented  prognostics -based 
control  routines  were  specified.  Arguments  were  provided 
for  the  potential  convenience  and  robustness  of  the  proposed 
notation  for  evaluating  prognostics-based  controllers. 
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